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METHODS, APPARATUS, COMPUTER PROGRAM AND STORAGE MEDIUM 

FOR POSITION DECODING 

Cross-reference to Related applications 

The present application claims the benefit of 
Swedish patent application No. 0301248-1, filed on April 
2 9, 2003, and U.S. Provisional patent application No. 
5 60/466,036, filed on April 29, 2003, which both are 
hereby incorporated by reference. 

Field of the Invention 

The present invention generally relates to decoding 
of position-coding patterns, and, more specifically, to 
methods for position decoding, in which a sequence of 

10 positions is extracted from a sequence of images of a 
position-coding pattern, as recorded by a sensor. 

The invention also relates to an apparatus, a 
computer program and a storage medium which can be. used 
to decode a position-coding pattern. 

15 Background Art 

Position-coding patterns are known which have the 
property that the absolute position in the position- 
coding pattern for a partial area of the position- 
coding pattern of a smallest predetermined size can be 

20 calculated by means of the information in the partial 
area and a predetermined algorithm. Examples of such 
position-coding patterns are disclosed in US 5,477,012 
(O. Sekendur) , where each position is coded by means 
of a symbol, and WO 92/17859 (Hewlett Packard Co.), 

25 US 6,570,104 (Anoto AB) , US 6,674,427 (Anoto AB) , 

US 6,667,695 (Anoto AB) , US 2003/0012455 (Anoto AB) and 
US 5,973,110 (Xerox Corp), where each position is coded 
by means of a plurality of symbols. 

Position-coding patterns of this type can be used, 

30 for instance, for digital recording of information which 
is written and/or drawn by hand by means of a user unit 
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on a base provided with the position-coding pattern. 
While moving the user unit, images of the position-coding 
pattern are continuously recorded locally at the tip of 
the user unit. One or more symbols in each of the images 
5 are decoded to a position. The decoded positions together 
constitute a digital description of the user unit's move- 
ment over the base . 

The position-coding pattern in the recorded images 
can be decoded in different ways. For instance, aforesaid 
US 2003/0012455 discloses a decoding method which 
involves calculating probabilities that symbols which are 
identified in a recorded area of the position-coding 
pattern assume any of a plurality of different possible 
decoding values. These probabilities are stored as real 
(floating precision) numbers in probability matrices and 
are used together with the different possible decoding 
values for decoding the identified symbols into 
coordinates for positions in the position-coding pattern. 

Aforesaid US 6,674,427 and US 6,667,695 disclose 
other ways of decoding or determining a position from a 
recorded image of a position-coding pattern. 

The known position decoding methods may be regarded 
to have the common features that symbol data is extracted 
from an isolated image in a sequence of images to be 
decoded; that position data is derived by effecting one 
or more lookup operations, using the symbol data, in one 
or more data structures that contain fundamental coding 
data of the position-coding pattern; and/or that the 
symbol or position data is used as input data to a 
mathematical formula for explicit calculation of an 
absolute position. 

Applicant's pending US application No. 10/282,001, 
which was not yet publicly available at the date of 
filing of the present application, but later published as 
US 2003/0128194, discloses a method for position 
decoding, where a position for a recorded area of the 
position-coding pattern is calculated in a known manner, 
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for instance in accordance with aforesaid US 
2003/0012455. However, in case a decoding error is 
detected for said position, either for the position as a 
whole or for one of its coordinates, an additional 
operation is performed as "a last resort"; information 
determined from the recorded area is compared or matched 
with information about how the position-coding pattern is 
composed in a matching area around a successfully decoded 
nearby position. The best matching portion of the known 
matching area is selected, and a corresponding position 
is determined as the decoded position. Thus, in summary, 
US 2003/0128194 discloses mandatory use of decoding by 
calculation of position from recorded area, followed, in 
case of a decoding failure, by an additional matching 
15 operation. 

Summary of the Invention 

An object of the invention is to provide an alterna- 
tive to known methods of position decoding from recorded 
images of a position-coding pattern. 

More specifically, an object of the invention is to 
allow fast, efficient and yet robust decoding of a 
position-coding pattern. 

Generally, the objects of the invention are at least 
partly achieved by means of methods, an apparatus, a 
computer program and a storage medium according to the 
25 independent claims, preferred embodiments being defined 
by the dependent claims. 

A first aspect of the invention is a method for 
position decoding, in which a sequence of positions is 
extracted from a sequence of images of a position-coding 
pattern, as recorded by a sensor. The method comprises 
the steps of: retrieving at least one reference position; 
and extracting said sequence of positions solely by 
matching information obtained from each of said images 
with a corresponding pattern reference area, which 
35 represents a known subset of the position-coding pattern 
with a given spatial relation to said reference position. 
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That said sequence of positions is extracted solely 
by matching information from each of said images with 
said corresponding pattern reference area means that the 
method is not based on any preceding step in which an 
attempt is made to calculate the positions in another way 
than through aforesaid extracting by matching. 

Each position in said sequence of positions may be 
extracted by: determining a relative location, within 
said pattern reference area, of a match between said 
information and said pattern reference area; and 
adjusting said spatial relation by said relative location 
to thereby derive said position. Thus, the sequence of 
positions may be extracted in a fast and efficient way. 
The pattern reference area, which represents the 
15 known subset of the position-coding pattern, may be 

generated for each image to be decoded. Alternatively, 
the pattern reference area may be generated only 
intermittently, to thus be used in the decoding of 
several consecutive images in the sequence of images. 

The given spatial relation of the pattern reference 
area to said at least one reference position may be 
either direct, in that the spatial location of each 
pattern reference area is directly defined or calculated 
from the reference position (s), or indirect, in that the 
spatial location is defined or calculated from one or 
more preceding positions, which are preceding in the 
sense that they have been extracted by preceding steps of 
matching. Each such preceding position again has a 
defined spatial location, directly or indirectly, with 
respect to the reference position (s) . 

The spatial relation may be given by a predicted 
position which is estimated based on said at least one 
reference position, and by said pattern reference area 
being generated with a given positional relationship to 
35 said predicted position. Preferably, the predicted 

position is included in the known subset of the position- 
coding pattern as represented by the pattern reference 
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area. Thereby, the matching may be effected around the 
predicted position. 

The step of estimating the predicted position may be 
effected for each image, to generate a sequence of 
predicted positions, which in turn may be converted, by 
said step of extracting by matching, to said sequence of 
positions. The conversion may be effected for a group of 
predicted positions, or sequentially by each predicted 
position being converted by matching to a decoded 
position before estimating a subsequent predicted 
position . 

Bach predicted position may be estimated based on at 
least two preceding positions selected from said at least 
one reference position and/or said sequence of positions. 

15 Moreover, each predicted position may be estimated by 
polynomial, such as linear, extrapolation of said at 
least two preceding positions. 

The reference position may be retrieved by 
calculating a position based on an image of said 

20 position-coding pattern in accordance with a prior art 
method of position decoding or calculation. Such a 
calculation may typically include: extracting symbol data 
from an isolated image preceding said sequence of images; 
deriving position data by effecting one or more lookup 

25 operations, using said symbol data, in one or more data 
structures that contain fundamental coding data of the 
position-coding pattern; and/or inputting said symbol or 
position data to a mathematical formula for explicit 
calculation of an absolute position. 

30 The ste P pf retrieving may be effected 

intermittently to update or replace said at least one 
reference position, the thus-retrieved reference 
position (s) being used in a subsequent step of extracting 
by matching for a subsequent sequence of images. The 

35 retrieval may be effected at a fixed sampling interval, 

or whenever a need for, or opportunity of, such retrieval 
is detected. In digitizing the movement of a writing 
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implement on a position-coded base, a single handwriting 
stroke may be represented by several consecutive 
sequences of positions, each being extracted by matching 
The method may suitably comprise the step of merging said 
at least one reference position with said sequence (s) of 
positions, so as to form a true representation of the 
movement . 

In one embodiment, the corresponding pattern refe- ■ 
rence area for an individual image among said sequence of 
images includes a plurality of partial areas defining a 
plurality of respective candidates to a position repre- 
sented by the position-coding pattern in said individual 
image. In this embodiment, the step of extracting by 
matching may include comparing the information obtained 
from said individual image with each of said plurality of 
partial areas included in the corresponding pattern 
reference area. The step of extracting may further 
include selecting the candidate for which the comparing 
step indicates correspondence between the information 
obtained from said individual image and any of said plu- 
rality of partial areas. 

The method described above for position decoding can 
advantageously be used together with any of the position- 
coding patterns that have been developed by Anoto AB and 
25 are described in for instance US 6,674,427 or 

US 6,667,695. Thus, the position-coding pattern may 
comprise a plurality of marks, wherein each mark codes 
one of at least two different values in at least one 
dimension. 

In one embodiment, which is adapted for use with a 
position-coding pattern of the above type, the step of 
extracting by matching includes obtaining said infor- 
mation from an individual image among said sequence of 
images by generating a probability matrix, said probabi- 
lity matrix representing a subarea of the position-coding 
pattern which is included in said individual image and 
containing one matrix element for each mark in said sub- 
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area, wherein each matrix element is adapted to store 
either a value which represents a most probable estimated 
value of its mark, or a value which represents that no 
value has been estimated for its mark. The value which 
5 represents a most probable estimated value is preferably 
an integer value. Using integer, e.g. binary, values in 
the probability matrix allows fast and efficient matching 
with the pattern reference area, since matrix operations 
performed on integers are faster to perform than for, 
10 e.g., real numbers. 

For a position-coding pattern of the above type, 
where each mark codes a binary value in a first dimension 
and a binary value in a second dimension, a first 
probability matrix may be generated for the values of the 
marks in said first dimension and a second probability 
matrix may be generated for the values of the marks in 
said second dimension. 

In one embodiment, said step of extracting by 
matching includes comparing the or each probability 
matrix with each of said plurality of partial areas 
included in the corresponding pattern reference area. 
This may be done by bit-wise correlation between the one- 
dimensional contents of the probability matrix and the 
contents, in the same dimension, of each equally sized 
25 partial area. 

Another embodiment is particularly adapted for use 
with a position-coding pattern, each dimension of which 
is based on shifts of a cyclic main number sequence, 
wherein any subsequence, of a first predetermined length 
or longer, of said cyclic main number sequence has an 
unambiguously determined position in said cyclic main 
number sequence, and based on a sequence of difference 
numbers representing differences in shifts between pairs 
of said main number sequence in the position-coding 
35 pattern, wherein any subsequence, of a second 

predetermined length or longer, of said difference number 
sequence has an unambiguously determined position in said 
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difference number sequence. This pattern may be of the 
type referred to above. In this embodiment, the step of 
extracting by matching includes: obtaining a first set of 
difference numbers for said pattern reference area; 
5 obtaining a second set of difference numbers from the or 
each probability matrix; matching said second set of 
difference numbers with said first set of difference 
numbers; and extracting a position of said position- 
coding pattern, in said dimension, from a match from said 
10 matching step. 

The second set of difference numbers may be obtained 
by matching respective rows or columns of the or each 
probability matrix with said main number sequence so as 
to estimate main number sequence positions for said rows 
15 or columns, and deriving the difference numbers of said 

second set of difference numbers by subtracting estimated 
main number sequence positions for pairs of rows or 
columns of the or each probability matrix. 

A second aspect of the invention is a method for 
position decoding by determining a position on the basis 
of information obtained from a partial area, recorded by 
a sensor, of a position-coding pattern, the position- 
coding pattern comprising a plurality of marks, each mark 
coding one of at least two different values in at least 
25 one dimension. This method includes the steps of: 

generating a probability matrix, said probability matrix 
containing one matrix element for each mark in said 
partial area, each matrix element being adapted to store 
either a value which represents a most probable estimated 
value of its mark, or a value which represents that no 
value has been estimated for its mark; matching said 
probability matrix with information about how the 
position-coding pattern is composed in a pattern 
reference area of said p'osition-coding pattern, said 
35 pattern reference area being larger than said partial 

area and including said partial area; and selecting, as 
the position to be determined, the one of the positions 
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defined by said pattern reference area for which the 
matching step indicates correspondence between said 
probability matrix and said matching area. 

The methods described above can be accomplished in 
5 hardware or software. Thus, a third aspect of the 

invention is a computer program, which comprises program 
code that, when executed in a computer, causes the 
computer to carry out a method according to the first or 
second aspect. 

10 The computer program can be executed by a processor 

which is arranged in the same unit as the sensor that 
images the partial area, for which a position is to be 
decoded, or in another unit. 

The computer program can be stored and distributed 

15 in the form of a storage medium, such as an electronic 
memory like RAM, SRAM, DRAM, SDRAM, ROM, PROM, EPROM, 
EE PROM, etc; an optical disk; a magnetic disk or tape; or 
some other commercially available storage medium. The 
storage medium can also be a propagating signal. 

20 According to a fourth aspect, the invention concerns 

an apparatus which comprises a signal-processing unit, 
which is adapted to perform a method according to the 
first or second aspect. 

The signal-processing unit may e.g. be realized 

25 by a suitably programmed processor, by specifically 

adapted hardware, such as an ASIC (Application-Specific 
Integrated Circuit) or an FPGA (Field Programmable Gate 
Array) , by digital or analog circuits or any combination 
thereof . 

30 Other objects, features and advantages of the in- 

vention are apparent from the following detailed de- 
scription of the invention, from the appended claims and 
from the drawings . 

Of course, features of the different embodiments of 

35 the invention may be combined in one and the same em- 
bodiment . 
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Generally, all terms used in the claims are to be 
interpreted according to their ordinary meaning in the 
technical field, unless explicitly defined otherwise 
herein. All references to "a/an/the [element, device, 
component, means, step, etc]" are to be interpreted 
openly as referring to at least one instance of said 
element, device, component, means, step, etc., unless 
explicitly stated otherwise. The steps of any method 
disclosed herein do not have to be performed in the exact 
order disclosed, unless explicitly stated otherwise. 
Brief Description of the Drawings 

The invention will now be described in more detail 
with reference to the accompanying drawings. 

Fig. 1 is a schematic view of a system in which the 
invention can be used. 
15 Fig. 2 is a schematic illustration of decoding and 

matching of partial areas of a position-coding pattern. 

Fig. 3 is a flow chart that illustrates a general 
method according to the invention of determining a 
position by decoding an image of a partial area of a 
20 position-coding pattern. 

Fig. 4 is a schematic illustration of a position- 
coding pattern which may be decoded according to the 
present invention. 

Fig. 5 shows four ideal locations of a mark in a 
25 position-coding pattern. 

Fig. 6 is a schematic illustration of an apparatus, 
in the form of an electronic pen, in which the position 
decoding according to the invention, may be carried out. 

Fig. 7 is a schematic block diagram of an electronic 
circuitry part and an image sensor in the device shown in 
Fig. 6. 

Fig. 8 is a schematic illustration that demonstrates 
the way in which a binary offset probability matrix is 
generated. 

35 Fi 9- 9 is a flowchart diagram of a first embodiment 

of a decoding method according to the invention. 
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Fig. 10 shows a generated local position-coding 
pattern and a binary offset probability matrix to illust- 
rate the operating principle of the first embodiment. 

Figs. 11, 12 and 13 are schematic illustrations to 
5 support the description of a second embodiment. 

Fig. 14 illustrates an alternative embodiment, where 
use is made of both the first and the second embodiment. 
Detailed Description of the Invention 

Fig. 1 shows a system for information management, in 
10 which the present invention can be used. The system com- 
prises a base 1 in the form of a sheet of paper, a user 
unit 2 and an external unit 3. The sheet of paper is 
provided with a position-coding pattern, only a small 
part 4 of which is schematically indicated in an enlarged 
15 scale. The user unit 2 can be used to write on the sheet 
of paper 1 in the same way as with an ordinary pen and to 
simultaneously record what is being written in digital 
form. The digitally recorded information can be processed 
in the user unit 2 and/or the external unit 3, to which 
it can be sent automatically (on-line) or when commanded 
by the user. For the digitally recorded information to 
correctly reflect the information on the sheet of paper, 
it is important that the position decoding be made as 
correctly and completely as possible. 
25 The Position-coding pattern, which will be described 

in more detail below, is made up of graphical symbols, 
which are arranged or designed according to predetermined 
rules. This means that if a partial area of the position- 
coding pattern is correctly imaged, said partial area 
having a predetermined minimum size, the position of the 
partial area in the position-coding pattern can be 
determined unambiguously by means of one or more symbols 
in the imaged partial area and a predetermined decoding 
algorithm. The position can be obtained as a pair of 
35 coordinates in a Cartesian or another coordinate system. 

When a user writes by means of the user unit 2 on 
the sheet of paper 1, the user unit will continuously 
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record images of a partial area of the position-coding 
pattern at the tip of the user unit. Each image is de- 
coded to a position. A sequence of such decoded positions 
then constitutes a digital representation of the movement 
of the user unit 2 over the sheet of paper and, thus, of 
what is written on the sheet of paper. 

The decoding of the position-coding pattern can take 
place in the user unit 2, in the external unit 3 or in 
some other unit to which the images of the position- 
coding pattern are sent in a more or less processed form. 

As mentioned above, the information in the partial 
area is decoded by means of a predetermined decoding 
algorithm. This functions well as long as the partial 
area is correctly imaged. Under actual conditions, how- 
15 ever, the user unit often cannot image the partial area 

entirely correctly, for instance because of poor lighting 
conditions, or when user holds the pen in an inclined 
position which causes the partial area to be imaged in 
perspective, or if the recorded image is deformed by the 
20 imaging mechanism or by other disturbances such as signal 
noise. It may happen that the decoding algorithm mis- 
interprets a symbol or misses a symbol, thereby making it 
harder to decode the position correctly. Errors can also 
arise if the position-coding pattern is not quite accura- 
25 tely reproduced on the sheet of paper but has been de- 
formed to some extent when being applied to the sheet of 
paper. Under certain conditions, errors can be dis- 
covered, for example, if the position-coding pattern con- 
tains redundant information which enables error detection 
30 or if the decoded position is unreasonable in relation to 
previously or subsequently decoded positions. 

Figs. 2 and 3 illustrate a general method according 
to the invention of determining a position by decoding an 
image of a partial area of a position-coding pattern. 
35 This method allows for fast, efficient and reliable 

decoding even if it has not been possible to successfully 
record and interpret all symbols within the partial area 
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of the position-coding pattern. A sheet of paper 1 is 
provided with a position-coding pattern P, which for the 
sake of clarity is not shown in Fig. 2. A small portion 
of the position-coding pattern P is however shown at 20 
5 in Fig. 4. As will be explained in more detail in the 
following, the position-coding pattern comprises a 
plurality of marks 22 which partly constitute the code 
symbols of the pattern. 

As seen in Fig. 2, two partial areas PA1 and PA1 • of 

10 the position-coding pattern P code positions PI and PI', 
respectively. It is assumed in Fig. 2 that positions PI 
and PI' have been successfully decoded from recorded 
images of partial areas PA1 and PA1 1 by way of any 
applicable decoding method. Next, in step 310 of Fig. 3 

15 an image of a new partial area PA2 (not explicitly shown 
in Fig. 2) is recorded, for which it is desired to decode 
a corresponding position P2 . The recorded image is pre- 
processed in a step 320 so as to form a suitable input 
for a following step 330, in which marks 22 are idehti- 

20 fied and a virtual raster 21 is fit to the marks. Steps 
320 and 330 may be performed by way of techniques that 
are well known in the art, such as filtering, binari- 
zation or thresholding, and utilization of the distance 
between different pairs of dots or utilization of Fourier 

25 transforms, as is described for instance in US 6,548,768 
and US 2002/0044138. 

Following steps 320 and 330, the actual decoding 
occurs in steps 340 and 350. First, a pair of matrices, 
referred to as x-BOPM and y-BOPM in the following, are 

30 generated by estimating the values represented by the 
marks 22 of the partial area PA2 in x and y dimensions. 
The information kept by these matrices will then be used 
for matching with information about how the position- 
coding pattern P is composed in a matching area MA. The 

35 information in the matching area MA may be generated 
exclusively for the decoding of only one position by 
matching. As seen in Fig. 2 and as will be described in 
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more detail later, the matching area MA is larger than 
the partial area PA2 and has a given positional relation 
to a prediction PA2 predicted of the partial area PA2 . The 
prediction PA2 predicted corresponds to a predicted position 
P2p red i C ted which is determined from the positions PI and 
PI* of the previously decoded partial areas PA1 and PA1 ' . 
Positions PI and PI- are thus examples of the "at least 
one reference position" referred to in the Summary 
section of this document, and the matching area MA is an 
example of the "pattern reference area". 

The size of the matching area MA may be set to a 
constant value which reflects a normal or default 
handwriting speed. For instance, the matching area MA may 
contain 32x32 code symbols. Alternatively, the size of 
the matching area MA may be adjusted dynamically 
depending on appropriate input data, such as current 
handwriting speed. The size of the partial area PA2 may 
be chosen in view of the size of the matching area MA and 
also depending on the siz.e of the recorded images. The 
partial area PA2 need not occupy the entire part of an 
recorded image. The partial area need not even have the 
above-discussed minimum size for unambiguous position 
determination. Generally speaking, the robustness of the 
decoding process increases with increasing size of the 
partial area, whereas the speed and the efficiency of the 
decoding process increase with decreasing size of the 
partial area. At present, a size of 16x16 code symbols is 
used for the partial area PA2 . 

The result of steps 340 and 350 may be the partial 
area PA2 matched that best matches an equally sized subarea 
of the matching area MA, and the relative location of 
PA2 matched with respect to the matching area MA and, thus, 
with respect to the predicted position P2 predicted . Hence/ a 
position P2 matched defined by the resulting partial area 
PA2 matched may be determined by simply adding this relative 
location to the predicted position P2 predicted . As seen in 
Fig. 2, while in practice the predicted partial area 



WO 2004/097723 



PCT/SE2004/000660 



10 



15 

PA2 pre dicted will not exactly correspond to the recorded 
partial area PA2, the matched partial area PA2 matched will. 
Thus, the result of the decoding method will be the 
position P2 matched which is identical to the position P2 
sought for. The determined position may be verified in a 
step 360. . 

A brief description of an apparatus - in the form 
of an electronic pen - in which the position decoding 
according to the invention may be carried out will now be 
given with reference to Figs. 6 and 7. 

The apparatus of Figs. 6 and 7 can constitute the 
user unit 2 in Fig. 1. It comprises a casing 2011 having 
approximately the same shape as a pen. In one short side 
of the casing there is an opening 2012. The short side is 
15 intended to abut against or- be placed a short distance 
from the surface on which the position determination is 
to take place. 

The casing essentially contains an optics part, an 
electronic circuitry part, and a power supply. 
20 The optics part comprises a light-emitting diode 

2013 for illuminating the surface which is to be imaged 
and a light-sensitive area sensor 2014, such as a CCD 
(Charge Coupled Device) or CMOS (Complementary Metal- 
Oxide Semiconductor) sensor, for recording a two- 
25 dimensional image. The apparatus may also comprise an 

optical system, such as a mirror and/or lens system. The 
light-emitting diode can be an infrared light-emitting 
diode and the sensor can be sensitive to infrared light. 

Generally speaking, the image sensor 2014 may be 
replaced by any kind of sensor that is suitable for 
imaging the position-coding pattern so that an image of 
the marks is obtained in black and white, in grey scale 
or in color. Such a sensor can.be a solid-state single- 
or multi-chip device which is sensitive to electro- 
magnetic radiation in any suitable wavelength range. For 
example, the sensor may include a CCD element, a CMOS 
element, or a CID element (Charge Injection Device). 
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Alternatively, the sensor may include a magnetic sensor 
array for detection of a magnetic property of the marks. 
Still further, the sensor may be designed to form an 
image of any chemical, acoustic, capacitive or inductive 
property of the marks. 

The power supply to the apparatus is obtained from a 
battery 2015 which is mounted in a separate compartment 
in the casing. The power supply can also be obtained via 
a cable from an external power source (not shown) . 

The electronic circuitry part 2016 comprises a 
processor unit with a processor which may be programmed 
to read images from the sensor and carry out position 
calculation, matching and verification on the basis of 
these images, as well as a working memory and a program 
15 memory. 

Furthermore, in the shown embodiment, the apparatus 
comprises a pen point 2017, by way of which it is 
possible to write ordinary pigment-based ink on the 
surface upon which the position determination is to be 
carried out. The pen point 2017 can be extendable and 
retractable, so that the user can control whether or not 
it is to be used. In certain applications, the apparatus 
need not have a pen point at all . 

Suitably, the pigment-based ink is of a type that is 
25 transparent to infrared light, while the marks absorb 
infrared light. By using a light-emitting diode which 
emits infrared light and a sensor which is sensitive to 
infrared light, the detection of the pattern takes place 
without any interference between the above-mentioned 
30 writing and the position-coding pattern. 

Moreover, the apparatus may comprise buttons 2018 
by means of which the user will activate and control the 
apparatus. It also comprises a transceiver 2019 for wired 
or wireless transmission, e.g. using cable, IR light, 
35 radio waves or ultrasound, of information to and from the 
apparatus. The apparatus can also comprise a display 2020 
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for showing positions, recorded information or other 
information to the user. 

The apparatus can be divided into different physical 
casings. The sensor and other components that are neces- 
sary for capturing images of the position-coding pattern 
and for transmitting them may be located in a first 
casing, while the signal-processing unit and the other 
components that are necessary for carrying out the 
position decoding may be located in a second casing. 

Fig. 7 is a schematic block diagram of the electro- 
nic circuitry part 2016 and the image sensor 2014 in the 
apparatus shown in Fig. 6. An ASIC 700 implements the 
electronic circuitry part 2016 and contains a CPU 720 as 
a main controller of the electronic circuitry. The CPU 
720 has access to a memory 730 through an on-chip data 
bus 7 60. The image sensor 2014 is implemented as a CMOS 
area sensor 750, which is connected to the data bus 760 
of ASIC 700 and which is capable of producing 96x96- 
pixel, 256-level grayscale digital images at a frequency 
of about 100 Hz. An image preprocessor 710 on ASIC 700 
will operate on images received from the image sensor 750 
in accordance with step 320 of Fig. 3 and store the 
preprocessed images in memory 730. The CPU 720 will 
generate the x-BOPM and y-BOPM matrices and will also 
perform the decoding of partial areas into positions. A 
matching unit 740 may assist the CPU 720 in these tasks. 
In an alternative embodiment, the ASIC 700 may be 
designed to operate without intermediate storage of the 
preprocessed images. 

In the following, the position decoding will be 
exemplified and described in more detail with reference 
to a specific position-coding pattern which has been 
developed by the applicant and which is thoroughly 
described in, inter alia, US 6,667,695. Position-coding 
35 patterns of similar types are described in detail in WO 
01/16691, US 6,663,008 and US 6,674,427. An alternative 
position-coding pattern is shown in US 6,570,104. 
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The position coding is based on a number sequence 
which below is referred to as difference number sequence. 
This difference number sequence has the property that if 
one takes an arbitrary partial sequence of a predetermin- 
ed length, for instance a partial sequence with five dif- 
ference numbers, this partial sequence always has an 
unambiguously determined location in the difference num- 
ber sequence. In other words, it appears only once in the 
difference number sequence and can thus be used for posi- 
tion determination. More specifically, the difference 
number sequence "runs" along the x axis as well as along 
the y axis of the position-coding pattern. 

The actual position-coding pattern is made up of 
simple graphical symbols 22 which can assume four diffe- 
15 rent values, 0-3. As seen in Figs. 4 and 5, each symbol 
22 contains a mark which, for instance, can have the 
shape of a dot and which can be displaced a distance 24 
in one of four different directions relative to a nominal 
position 23 or raster point. The nominal position 23 is 
20 at an intersection between raster lines 21 in a grid that 
may be virtual, i.e. invisible to the area sensor in the 
pen, or actual, by the raster lines being wholly or 
partly visible to the area sensor. The value of the 
symbol is determined by the direction of the dis- 
25 placement. Each symbol value 0-3 can be converted into 

one bit which is used for coding an x coordinate and one 
bit which is used for coding a y coordinate, i.e. into 
the pairs of bits (0,0), (0,1), (1,0), and (1,1). Thus, 
the coding is made separately in' the x direction and the 
y direction, but the graphical coding is made with a 
symbol which is common to the x and y bits. 

The distance 24 is suitably not less than about 1/8 
and not more than about 1/4, preferably about 1/6, of the 
distance between two adjacent raster lines 21. The 
35 distance can, for example, be about 300 micrometers or 
about 254 micrometers. The latter distance is particu- 
larly suitable for printers and scanners, which often 
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have a resolution that is a multiple of 100 dpi (dots per 
inch) . 

Each position is coded by 6 * 6 symbols which can 
thus be converted into a 6 * 6 bit matrix for an x coor- 
5 dinate and a 6 * 6 bit matrix for a y coordinate for the 
position. If the x bit matrix is considered, this can be 
divided into six columns of six bits each. Each sequence . 
of bits in a column constitutes a partial sequence in a 
63-bit-long cyclic main number sequence which has the 
property that if a partial sequence having a length of 
six bits is selected, this has an unambiguously deter- 
mined place in the main number sequence. The six columns 
can thus be converted into six position numbers or 
sequence values which correspond to six places in the 
15 main number sequence. Between these six position numbers, 
five difference numbers can be formed in pairs, which 
constitute a partial sequence of the difference number 
sequence and which thus have an unambiguously determined 
place in the same and, consequently, an unambiguously 
determined place along the x axis. For a certain x co- 
ordinate, the position numbers will vary depending on the 
y coordinate. On the other hand, the difference numbers 
will be the same independently of the y coordinate, since 
the position numbers always vary according to the main 
number sequence, which is repeated cyclically in the 
columns in the entire position-coding pattern. 

Correspondingly, six rows in the y bit matrix define 
six position numbers in the main number sequence. These 
six position numbers define five difference numbers, 
which constitute a partial sequence of the difference 
number sequence and which thus have an unambiguously 
determined place along the y axis. 

When the position-coding pattern is used, images 
of different parts of the position-coding pattern are 
35 recorded, as has been described above. As a rule, the 
images contain considerably more than 6*6 symbols. 
Thus, the decoding may be based on the best n * m symbols 
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in the image, with n and m being any number of symbols 
equal to 6 or larger. For example, the decoding may be 
based on the best 8 * 8 or 16 * 16 symbols. The extra 
symbols are used, inter alia, for error detection and/or 
correction. 

In the following, position decoding based on the 
above-described position-coding pattern will be described 
with reference to the drawings. The position decoding is 
preferably carried out by means of a processor and suit- 
able program code which can be available in the user unit 
2 or the external unit 3 in Fig. 1 or in some other unit. 

Since the position-coding pattern is not always per- 
fectly reproduced on a surface and since the user unit 
sometimes deforms the position-coding pattern during 
imaging, it may be difficult to determine quite safely 
the displacements of the dots and, thus, the value of the 
symbols. Instead, for each symbol the probability that 
the symbol assumes any of the four different possible 
values may be determined. According to embodiments of the 
present invention, aforesaid two matrices are generated 
in step 340 of Fig. 3 for an imaged partial area PA - one 
for columns (x coordinate encoding) and one for rows (y 
coordinate encoding) . Each matrix element in each of 
these two matrices may contain an integer value that 
25 represents the estimated binary value of the symbol in 

the x direction and y direction, respectively. This is in 
contrast to the prior art, e.g. aforesaid 
US 2003/0012455, where each matrix element contained two 
real numbers (floating-precision numbers) representing 
30 the probability of the symbol value "0" and "1", 

respectively. These two matrices may be referred to as 
Binary Offset Probability Matrices (BOPM) and are 
identified as "x-BOPM" and "y-BOPM" in the drawings. 

In addition, each matrix element in the x-BOPM and 
y-BOPM matrices may indicate that no binary value was 
estimated for the symbol, e.g. because it was not 
possible, at some level of certainty, to decide which of 
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the four possible displaced positions (Fig. 5) that the 
mark 22 in question assumes. 

Thus, each matrix element in the x-BOPM and y-BOPM 
matrices will contain any of the values "0", "1" or 
5 the latter representing that no binary value was esti- 
mated for the symbol . 

Various methods may be employed when analyzing a 
recorded and preprocessed image so as to estimate, if 
possible, the binary values of the symbol in the x and y 
10 directions - as given by the displacement of the mark 22 
- and otherwise represent the symbol by the uncertainty 
value 

For instance, for each identified mark it may be 
investigated whether it lies within a certain "safe" zone 

15 around any of the four ideal displacement positions shown 
in Fig. 5. Such "safe" zones may e.g. be defined by 
circular areas having their centers at respective ones of 
the four ideal displacement positions and having a 
predefined radius. Thus, if a mark is found to lie within 

20 any such "safe" zone, the symbol is held to have the 

value represented by the ideal displacement position that 
forms the center of the "safe" zone. 

Alternatively, "full-scale", real-number probability 
matrices may be calculated based on a distance function 

25 that outputs a real-number probability value based on the 
displacement of a mark from an ideal displacement 
position. Examples of such calculations are given in 
aforesaid US 2003/0012455. Then, probability threshold 
values may be applied to the real-number probability 

30 matrices, such that if for instance such a probability 
matrix indicates that a certain mark encodes a "0" or a 
"1" at a probability in excess of 0.8, where 0.8 is the 
threshold, then the corresponding matrix element in the 
x-BOPM or y-BOPM matrix will be set to "0" or "1"; 

35 otherwise the matrix element will be set to "-". 

As still an alternative, if the encoding of the 
marks is such that the same x value (y value) is used for 
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the two ideal positions that lie above a diagonal line 
through the raster point 23 {e.g. positions b and c in 
Fig. 5) as well as for the other two that lie below such 
a diagonal line (e.g. positions a and d in Fig. 5), the 
location of an identified mark and its distance to such a 
diagonal line may be used to estimate the binary values 
of the symbol in question. 

The x-BOPM and y-BOPM matrices thus produced are 
used in step 350 of Fig. 3 in two different ways, 
according to the disclosed embodiments, so as to decode 
the position coded by the pattern in the partial area PA 
of the recorded image. 

In a first embodiment, which will be described with 
reference to Figs. 8-10, a method referred to as Pattern 
15 Neighborhood Matching (PNM) is used for matching a re- 
corded partial area PA (as represented by the x-BOPM and 
y-BOPM matrices) with a larger matching area MA of the 
position-coding pattern, generated around a predicted 
position in the position-coding pattern. The predicted 
20 position is produced from preceding positions defined by 
preceding partial areas of the position-coding pattern. 
Here, a complete bit-wise correlation between the 
recorded partial area PA and the generated matching area 
MA is performed. Thus, this method will be capable of 
25 successful decoding also if the recorded partial area PA 
contains several uncertain or incorrectly estimated 
symbols . 

A second embodiment, which will be described with 
reference to Fig. 11-13, concerns a method referred to as 
Pattern Sequence Matching (PSM) . Here, no full bitwise 
correlation between recorded partial area PA and matching 
area MA is performed. Instead, matching is made between 
difference subsequences found in the recorded partial 
area PA and difference subsequences for a larger matching 
35 area MA, which again is generated around a predicted 

position in the position-coding pattern. This method will 
be faster, since the number of matches performed is 
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smaller than for the PNM method (ideally, the running 
time will be proportional to the sum of the side lengths 
of the matching area as compared to the product of the 
side lengths) . It i S/ however, less capable of decoding 
partial areas with many uncertain or incorrectly 
estimated symbols. 
Pattern Neighborhood Matching 

As explained above, the position-coding pattern is a 
composition of an x encoding and a y encoding. Thus, the 
decoding may be handled separately for the two dimen- 
sions. Fig. 8 schematically shows a recorded partial area 
800, consisting of 10x10 raster intersections and dots 
positioned at different displacements from the respective 
intersections. For each column (and each row) some of the 
encoded x (and y, respectively) values may be determined 
by identifying that a dot is located sufficiently close 
to one of the four possible ideal displacements - 
"right", "upwards", "left" or "downwards" by e.g. 1/6 of 
the raster cell size. For instance, dot 840 may be" 
determined to represent displacement "left", thereby 
yielding (x, y) = (l, 0) according to coding legend 830. 
On the other hand, at some raster intersections it will 
not be possible to decide what a nearby dot, if any, 
encodes. For instance, dot 850 is located too close to 
its raster intersection, and at intersection 860, no dot 
is present at all. in these cases, neither x nor y value 
can be estimated. Thus, there is a need to express un- 
certainty about the value at a certain raster inter- 
section, in x and/or y dimension. 

By traversing the columns and rows of the recorded 
partial area in the above-described manner and in 
accordance with step 340 in Fig. 3, three characters or 
values "0", "1" and "-" may be used to represent the 
interpretation of column encodings (e.g. 820 in Fig. 8) 
and row encodings (e.g. 810 in Fig. 8), wherein the "-" 
indicates uncertainty about the dot displacement. 
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As already mentioned, this process results in two 
matrices, one for columns (x-coordinate encoding) and one 
for rows (y-coordinate encoding), namely the binary 
offset probability matrices x-BOPM and y-BOPM. An y-BOPM 
example is given at 1010 in FIG 10. In a sense, the x- 
BOPM and y-BOPM are trinary rather than binary, since in 
addition to binary values they also have one value that 
represents uncertainty. 

The x-BOPM and y-BOPM may be stored in memory 730. 
In more detail, each row of a matrix may be stored in a 
first n-bit data register (n being the size of the matrix 
in one dimension) that represents the "certain" values 
"0" and "1", and in a second n-bit data register that 
represents bit flags as to whether or not uncertainty 
15 applies to the corresponding value in the first n-bit 
data register. If a bit flag is set in the second data 
register, this will mean that the corresponding bit value 
is not to be used and that the matrix cell in question 
has a "-" value. 

20 Fig. 9 illustrates the pattern neighborhood matching 

method 900, which implements the decoding and verifying 
steps 350 and 360 for this embodiment. Input data for 
this method is the above-mentioned two binary offset 
probability matrices x-BOPM and y-BOPM, as generated in 
25 step 340 of Fig. 3, as well as the two preceding posi- 
tions PI and PI' of Fig. 2. 

In step 910, based on the two preceding positions PI 
and PI', a next position P2 predicted i s derived by linear 
extrapolation from positions PI and PI'. As an alterna- 
tive, more than two preceding positions may be used for 
predicting the next position, wherein any known motion 
prediction model may be used, such as polynomial extra- 
polation. As yet an alternative, only one preceding 
position may be used, if the prediction is combined with 
35 motional data input concerning e.g. direction of move- 
ment, speed or acceleration of the user unit 2, or other 
useful information, such as default or predefined in- 
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formation on expected handwriting speed. Only one 
preceding position may also be used if the size of the 
matching area MA is set adequately large to include a 
following position. Such a matching area is suitably 
centered around the preceding position. The above 
alternatives may also be combined, e.g. by different 
numbers of preceding positions being used at different 
stages of the decoding process. 

The or each preceding position PI, pi » may be 
determined in different ways. For instance, at the very 
beginning of a pen stroke, PI, Pi' may be calculated in 
accordance with any prior art method, including but not 
limited to the ones described in aforesaid US 6,667 695 
US 2003/0012455 and US 6,674,427, all incorporated 
15 herewith by reference. Then, as new positions have been 
decoded by matching in accordance with the invention, 
they may be used as preceding positions PI, PI' to form 
the basis of the prediction of a new position P2 . 

In step 920, the matching area MA (Fig. 2) of the 
position-coding pattern is generated around the predicted 
position. An example of a generated matching area is 
given at 1000 in Fig. 10. For instance, aforesaid 
US 6,667,695, in column 11, line 57 - column 15, line 14, 
and in Fig. 6 thereof, describes in detail how an area of 
the position-coding pattern may be generated from a given 
position. 

In step 930, one of the binary offset probability 
matrices, e.g. y-BOPM, is matched with the contents of 
the generated matching area in the corresponding 

30 dimension, e.g. y dimension. To this end, matching unit 
740 will perform bitwise correlation between the y-BOPM 
and each equally sized partial area of the matching area. 
For each such bitwise correlation, the matching unit 740 
will count the number of corresponding bits in the y-BOPM 

35 and the partial area. If a bit is set to the uncertainty 
value "-» in the y-BOPM, this position will simply be 
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dispensed with and no bit correspondence will of course 
be credited this position. 

Once the y-BOPM has been matched with the entire 
matching area, the CPU 720 will determine the one of the 
partial areas for which the best match occurred with the 
y-BOPM (cf PA 2matched in Fig. 2) . m Fig. 10, the best 
matching partial area of the matching area 1000 is indi- 
cated at 1020. 

In step 940, a resulting position (cf P2 matched in 
Fig. 2) is calculated from the best matching partial 
area. This position is easily determined, since the 
absolute start position of the matching area MA in the 
position-coding pattern is already known (given by the 
predicted position P2 predicted ) . Thus, the resulting posi- 
tion is added as a relative position to the known abso- 
lute start position of the matching area MA so as to 
yield absolute-value coordinates of the predicted posi- 
tion. 

It is to be observed that this embodiment results in 
a fully decoded position (both x and y coordinates) from 
the row pattern on y-BOPM matching. Nevertheless, by also 
performing x-BOPM matching with the matching area in 
steps 950 and 960, the result of steps 930 and 940 may be 
verified to gain robustness: if and only if the best 
correlations agree on both coordinates, the result is 
accepted in step 970. Thus, steps 950 and 960 may imple- 
ment the verifying step 360 of Fig. 3. 

As an alternative to such a separate . verifying 
operation in steps 950 and 960, two match matrices, one 
for the y-BOPM matching and one for the x-BOPM matching, 
may be generated in parallel in steps 930 and 940. A 
matrix element in each of these match matrices will 
represent the number of matching x/y bit values for the 
partial area PA that starts with that matrix element's 
position in the matching area MA. By adding these two 
match matrices element by element, the CPU 720 may select 
the particular element position having the highest 
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resulting sum as the matched position P2 matched sought for. 
Thus, in this case a verification is inherent in the 
matching operation. 

Alternatively or additionally to the above 
5 verification methods, the partial area(s) with the best 
correlation with the x-BPOM and/or the y-BOPM may only be 
used for position determination if a matching criterion 
is met. The matching criterion may include that the 
number of matches (i.e. bit correspondences) exceeds a 

10 predetermined threshold value and/or that the ratio of 
the number of matches to the number of possible matches 
exceeds a predetermined threshold value. The number of 
possible matches may be given by the number of "certain" 
values (i.e. not set to the uncertainty value "-") in the 

15 x-BOPM and y-BOPM, respectively. 
Pattern Sequence Matching 

This method makes use of the fact that positions are 
coded in the position-coding pattern based on two one- 
dimensional difference number sequences. Thus, the " 

20 matching may be handled as two one-dimensional problems 
rather than one two-dimensional. This will mean the 
searching time required for matching is linear (ordo 
(n+m) ) instead of quadratic {ordo (n*m) ) , n and m being 
the side lengths of the matching area. 

25 With reference to Fig. 13, the pattern sequence 

matching (PSM) method 1300 starts with a step 1310 of 
predicting the next position P2 preclicted . step 1310 may be 
performed in the same way as has been described above for 
step 910 of Fig. 9. 

30 In a ste P 1320, matching difference number sub- 

sequences MSx and MSy (see 1220 in Fig. 12) are generated 
for a matching area MA (see 1200 in Fig. 12) around the 
predicted position P2 pre dicted. The actual matching area 
itself is not generated, though, in contrast to the first 

35 embodiment. Aforesaid US 6,667,695 describes, in column 
11, line 57 - column 15, line 14, and in Fig. 6 thereof, 
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how a difference number subsequence can be calculated 
from a given position. 

A step 1330 derives sets of difference numbers Sx 
and Sy for the recorded partial area PA2, to be matched 
with the predicted matching difference number 
subsequences MSx and MSy produced in step 1320. First, 
with reference to Fig. 11, for each row/column 1110 in 
the y-BOPM/x-BOPM the best matching main number sub- 
sequence 1120 in the main number sequence 1100 is deter- 
mined. This may be done by a fast matching algorithm, 
based on the fact that if six consecutive accurate bit 
values (i.e. having estimated values "0" or "1" but not 
"-") are found in the BOPM, this combination of six 
values will inherently have a unique position in the main 
15 number sequence. This unique main number sequence 

position may be obtained directly from a lookup table 
which is conveniently stored in memory 730 and consists 
of the 63 different cyclic shifts of the main number 
sequence 1100. The other accurate positions in the 
particular row/column 1110 of the y-BOPM/x-BOPM are then 
checked against the particular main number sequence shift 
from the lookup table, and if also these are found to 
match, then the derived main number sequence position is 
held to be correct. If not, a full bit-wise matching is 
25 done between the particular row/column 1110 of the y- 

BOPM/x-BOPM and all 63 main number sequence shifts in the 
lookup table, wherein the best match is selected as the 
derived main number sequence position for that row/column 
1110. Simultaneously, a main number sequence match rate 
30 is registered in the form of the total number of matching 
bit values for the particular row/column 1110 of the y- 
BOPM/x-BOPM. 

Then, step 1330 continues by deriving sets of 
difference numbers Sx and Sy (1230 in Fig. 12) by sub- 
35 tracting derived main number sequence positions for ad- 
jacent rows/columns. Also, each difference number is 
assigned a difference number match rate, for example by 
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weighting (e.g. summing or averaging) the main number 
sequence match rates of the related rows/columns, or by 
selecting the minimum thereof. 

Now, in step 1340 the thus derived (i.e. observed) 
sets of difference numbers Sx and Sy are matched with the 
predicted difference number subsequences MSx and MSy, as 
indicated at 1220 in Fig. 12. The best match in the 
subsequences MSx and MSy may be selected as the 
respective consecutive combination of difference numbers 
that yields the highest number of matches, optionally 
further distinguished by selecting the combination 
thereof having the highest sum of difference number match 
rates. Alternatively, the best match may be selected as 
the combination of matching difference numbers that 
15 yields the highest sum of difference number match rates, 
irrespective of the number of matches. 

In step 1350 the resulting position P2 matche d is 
determined from the resulting best match in step 1340. 
For reasons that are similar to the ones given in the 
description above for step 940 of Fig. 9, this resulting 
position is easily calculated. 

In step 1360, the resulting position P2 mat ched is 
verified, for instance by generating a local verifying 
area of the position-coding pattern around position 
25 P2 mat ched and correlating this bit-wise with the x-BOPM and 
y-BOPM. This verification may include a matching 
criterion. As with the above-described Pattern 
Neighborhood Matching, the matching criterion may include 
that the number of matches (i.e. bit correspondences) 
should exceed a predetermined threshold value and/or that 
the ratio of the number of matches to the number of 
possible matches should exceed a predetermined threshold 
value. Instead of generating a complete local verifying 
area around position P2 matched , use may be made of the 
35 calculations performed in step 1330. Since the selected 
resulting position P2 mat ched is the one with the best 
match, many of the derived difference numbers are likely 
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to correspond with the predicted ones. The numbers of 
matches for the best matching main number sequences 
underlying these corresponding difference numbers are 
already known, via the above-mentioned main number 
sequence match rate. Thus, a local verifying area need 
only be generated for difference numbers which do hot 
match . 

Combination of Pattern Sequence Matching and Pattern 
Neighborhood Matching 

The two embodiments may be combined, for instance as 
will be described with reference to Fig. 14. 

As seen at 1400, matching is performed based on the 
generated x-BOPM and y-BOPM and the matching area 
information obtained through the predicted position. 
15 First, pattern sequence matching 1402 is performed. If 

the decoding is successful, verification is performed at 
1410. If, on the other hand, the pattern sequence 
matching 14 02 fails, then pattern neighborhood matching 
1404 is performed. If this succeeds, verification follows 
20 at 1410. However, if also the pattern neighborhood 

matching 1404 fails, an attempt to solve the position by 
other, known methods is made at 1420. Such methods may 
involve any of the ones described in aforesaid 
US 6,667,695, US 2003/0012455 and US 6,674,427. There are 
25 situations where one could expect such methods to succeed 
even if the matching methods according to the invention 
have failed. One example is if the size of the matching 
area MA is too small to "catch up with" a fast-writing 
user. Another example is when the pen is moved over a 
30 position discontinuity in the position-coding pattern, 
for instance when a pen stroke is drawn from one 
position-coded base to another. If the position is 
successfully solved at 1420, the result is verified at 
1410. 

35 If verification of a result from matching 1400 

fails, a further attempt may be made to solve the 
position at 1420. If also this attempt fails, another 
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existing position-solving method may be tried at 1420, 
until all available ones have been tried. 

The invention has mainly been described above with 
reference to a few embodiments. However, as is readily 
appreciated by a person skilled in the art, other embodi- 
ments than the ones disclosed above are equally possible 
within the scope of the invention, as defined by the 
appended patent claims . 
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